<P> <img src="https://i.ibb.co/gyNf19D/nhslogo.png" alt="nhslogo" border="0" width="100" align="right"><font size="6"><b> CS4132 Data Analytics</b> </font>

# Crosswords: Accessibility and Representation

### by Chua Wee Chong

<div class="alert alert-block alert-danger">

Important Note: Please keep your report concise and relevant (i.e. show only relevant steps and visualizations used to answer your research questions).

</div>

# Table of Content (with relevant hyperlinks to sections)

1. [Motivation & Background](#motivbg)  

2. [Research Questions and Summary of Answers](#qns)  

3. [Datasets](#dataset)

4. [Methodology](#method)

5. [Data Acquisition](#acquire)  

6. [Data Cleaning](#cleaning)  

7. [EDA](#eda)  

8. [Results Findings & Conclusion](#results)  

9. [Recommendations or Further Works](#recommendations)  

10. [References](#ref) 

​

# Motivation and Background<a id='motivbg'/>

<div class="alert alert-block alert-warning">

Give an overview of the project, motivation, background and goals.

</div>

<p> American crosswords are word puzzles, in which the goal is to fill in all the white squares with letters to fit the given clues. The special rules of American crosswords in particular are that each square must be used twice and each word has to be at least 3 letters long. The answers are a mix of trivia and common phrases. There are two main criticisms of American crosswords. </p>

<p>

The first criticism is how accessible it is. Sometimes, obscure terms must be used to fill the grid, as the constructor cannot find a better configuration. These obscure terms are called "crosswordese". They appear commonly in puzzles as they have convienent letter patterns. However, as computers have evolved to assist humans in construction, the quality of puzzles have been getting better and better. </p>

<p>The second criticism is how representative it is. Crosswords have been said to reflect a piece of personality from the contructor. Originally, crosswords were for straight liberally-educated white men. As time passed, people realised that there was a lack of representation in terms of answers or clue-writing for the other groups. Hence, there has been a push to include more of these people into the crossword. This includes mentorship to women/people of colour/LGBTQ people to construct crosswords. </p>

<p>

This project would aim to find how accessible the crossword currently is, given the rise of computers as a construction aid. In a similar fashion, it would also like to find out how representative the crossword is of minorities. </p> 

# Summary of Research Questions & Results<a id='qns'/>

​

<div class="alert alert-block alert-warning">

Repeat your research questions in a numbered list. After each research question, clearly state the answer/conclusion you determined. Do not give details or justifications yet — just the answer

</div>

Accessibility:

<p>1. <b>"Crosswordese"</b> 

​

How has the amount of obscure answers changed throughout the years? Crosswordese is the use of an obscure word with a convienent letter pattern, with many common letters or vowels sometimes to fill in the grid. These words make it hard for people to do them, if they are not part of an "in-group" that knows all these common crossword words. Hence, I would like to find out, how much "crosswordese" the crossword has over the years.</p>

- Crosswordese has only somewhat decreased through the years, not a very significant change.

- Crosswordese slightly increases throughout the week, making it less accessible for newbies.

<p>

2. <b>Freshness</b>

​

How has the "freshness" factor of crossword changed over the years? Crosswords are a reflection of the world, what the

current trends are and such.  With more and more crosswords in the pool, the number of never seen before terms and names in the crossword has steadily decreased. However, words and phrases get coined every single day, some catching on in modern language. This question aims to investigate that, coupled with how computers have helped give more liberty to filling the grid. </p> 

- Freshness has increased throughout the years, giving a better puzzle.

- Freshness increases through the week, as puzzles become more loose in theme. <p>

Representation:

<p>3. <b>Inclusive Clues</b>

​

How has the clue-writing changed over the years? Clue-writing is half of the puzzle. This could introduce some unwanted sterotypes. For example, the clue-writing for the answer MIT has been associated with males more than females, showing a bias in thinking that men are more prominent in tech. Hence, by finding out the mentions of minorities in clue-writing, one can find out how progressive the puzzle has become.</p>

- Female names are used less than male names in clues and answers combined.

- However, the use of names has also decreased through the years.

- Some outlets strive to make female names used as much as male names, and a shift to equality has been seen in recent years. <p>

<p>4. <b>Constructors</b>

​

How has the make-up of constructors changed over the years, and modified the quality of crosswords? As said, a crossword reflects a person's experiences and views. With more diverse make-up of constructors, there will be more variety for that. There was a time where the constructors were mostly men, skewing the quality for some. With the rising number of mentorships given to minorities by prolific constructors, however, there has been an uptick in minority constructors. This question would like to analyse the trend as a whole, as well as possibly see the impact of mentorship. </p>

- Women and new constructors are seen to mostly construct only early-week puzzles.

- Female representation in construction has positive impacts on inclusivity.

- Collaborations boost the quality of crosswords for all. <p>

# Datasets<a id='dataset'/>

<div class="alert alert-block alert-warning">

Numbered list of dataset (with downloadable links) and a brief but clear description of each dataset used. Draw reference to the numbering when describing methodology (data cleaning and analysis).

</div>

1. https://www.crosswordgiant.com/browse (website to scrape for the clue answer pairs)

2. https://www.xwordinfo.com/ (contains more data about the NYT Crossword in particular)

3. https://books.google.com/ngrams/ (for word searching)

4. https://peterbroda.me/crosswords/wordlist/lists/peter-broda-wordlist__scored.txt (crossword construction wordlist by Peter Broda)

5. https://drive.google.com/uc?export=download&id=1Ruxn8XzRNstU6sDPOMm_K72fVookrPPr (crossword construction wordlist by Brooke Husic and Enrique Henestroza Anguiano)

6. https://www.verywellfamily.com/top-1000-baby-boy-names-2757618 (boy name list)

7. https://www.verywellfamily.com/top-1000-baby-girl-names-2757832 (girl name list)

# Methodology<a id='method'/>

<div class="alert alert-block alert-warning">

You should demonstrate the data science life cycle here (from data acquisition to cleaning to EDA and analysis etc).

</div>

## Data Acquisition<a id='acquire'/>

<div class="alert alert-block alert-info">

Display the data which will be used in the project. 

The data should be saved in .xlsx or .csv format to be submitted with the project. If webscraping has been done to obtain your data, save your webscraping code in another jupyter notebook as appendix to be submitted separately from the report. 

Import and display each dataset in a dataframe. 

For each dataset, give a brief overview of the data it contains, and explain the meaning of columns that are relevant to the project.

</div>

Many of these datasets were scraped from the internet. The scraping code can be found in the Appendix.

### CrosswordGiant.com

</p>This data was scraped from CrosswordGiant.com, looking at all the pages, then filtering and sorting the relevant publication outlets. The 5 DataFrames below are very similar, they differ only in the outlet column. Inside each DataFrame, there are 4 columns. <p>

</p>Clue - What hint was given to the answer in this crossword? <p>

</p>Answer - What is the expected answer to the given clue?  <p>

</p>Outlet - Where was the crossword published?  <p>

</p>Date - When was the crossword published?<p>

### XWordInfo.com

This dataset is obtained from XWordInfo, a site with extensive information on New York Times Crosswords. The dataset is named NYTCI, standing for New York Times Constructor Info. The dataset has 8 columns. First two are the Day Of Week and Date. Some crosswords made are collaborations, made with up to 3 people, hence C1,C2 and C3, which stand for Constructor 1, 2 and 3 respectively. For some crosswords, there are less than 3 constructors, hence their columns are dashed. C1,C2,C3 No. stand for the order of the puzzle the author has published up til now. C1, C2, C3 Gender stand for the genders of Constructors. 

### Google NGram

Taking the some of the most common answers over all crosswords, around 15K, we input it into Google NGram as an URL, which gives us scrapable data. We then take the relevant years of our crosswords, 1990-2020, and insert it into a DataFrame.

However, sometimes the data is missing, for example I found no data for "ISNT". Since this is uncommon and shows no pattern that can be seen, it is reasonable to assume randomness and just ignore it.

Here, there are 3 different files as some answers were scraped in different sessions

The resulting DataFrame is displayed. 

### Crossword Wordlists

<p> In crossword construction, wordlists are used. These wordlists are fed into a program, which will help suggest the best configuration for a particular section, or even for the whole grid. As such, the wordlists are as comprehensive as possible, trying to maximise the number of configurations to pick from, to pick the best one to human eyes. These wordlists are usually scored by the author as well, giving a score of how good, in their opinion, an answer is.

Using these wordlists, we can also check how "good" each crossword is, giving a quantifiable amount of weight to each asnwer. <p>

Of course, these wordlists are biased based on who makes it. Hence, we shall try to use 2 independent wordlists to cross-check. There are many wordlists out there, however, these two are chosen as they are very comprehensive, but also free. 

### Girl's/Boy's Name Database

This dataset was acquired by simply going to the website and doing a copy-paste.

This database will be used in Q3 to look for occurrences of their names. It will only be used as a lookup, not really a dataframe.

## Data Cleaning<a id='cleaning'/>

<div class="alert alert-block alert-info">

For data cleaning, be clear in which dataset (or variables) are used, what has been done for missing data, how was merging performed, explanation of data transformation (if any).

If data is calculated or summarized from the raw dataset, explain the rationale and steps clearly.

</div>

Since most of the data is scraped, I have been able to control the cleaniness of data, therefore, the quality and cleaniness of the data was high. Of course there were some hitches during the data collection. Missing data is rare and may not exist in the dataset.

​

For two of the datasets, CrosswordGiant and NGram, there was the possibility of the data not existing. This was simply handled by catching the exception/error that occurs when I tried to process the empty data. Hence, it is ensured that no data that is invalid is entered into the saved file.

Again, the code is found in the Appendix.

​

For the namelist, no cleaning is required; that has already been done by the publisher.

For checking symmetry, the dataset is very simple and acquired by scraping. Although some of the entries are incorrect, they are at random. This was caused by a logic error that I did not have the skills to fix. However, this should not affect the results significantly. However, no cleaning is requried.

### CrosswordGiant

<p>For this dataset in particular, some webpages have garbage data, with the answers just being XXXXXXXX or being duplicated many times. The former is harder to detect, and can be cleaned later, when answering question 1. The latter can easily be removed by checking how many entries the crossword of the particular day has, then just removing them.<p>

Sometimes, publications fill their crosswords with puns, which are bogus words, without a theme.

These gimmicks are hard to detect, and unfortunately CrosswordGiant is unable to detect such cases.

This problem is difficult to solve, as it is a linguistical one, and not within the scope of this project.

Some introduction is required here. Bogus words follow a theme, and themed crosswords appear on certain days of the week only. By and large, it is reasonable to assume that bogus words appear at random, and are independent between crosswords.

​

Hence, these will be the steps for cleaning this dataset.

1. Given the date, find the day of the week.

2. Toss out known puzzles with bogus words.

3. Find the puzzles with too many clues and discard them.

Then, we just merge them.

​

While doing the project, I found that this dataset was missing some New York Times crossword from around 2000. This missing it is not crucial to the project, and hence can be ignored.

### XWordInfo

Curiously, their formatting is sometimes irregular. The numbering system calls the first puzzle "the debut puzzle" and others "puzzle # n". For puzzle number, this is an easy fix. They also call people Mr or Ms, depending on their gender. This was easy to replace. The harder part however, was the inconsistencies in their data. For some people, Their name was used instead of Mr X or Ms X. Since those are very few, I have taken the step to clean it by hand.

​

There is a person's gender as "A". This arose due to how the scraping was done. Visiting the website, I have found the persons name and found out that their name is male.

### Google NGram

The interesting thing about Google NGram is that it returns different results based on the capitalisation of the word. Hence, I tried both all caps and no caps form of the word.

This has yielded a DataFrame with two of the same words. This is largely easy to clean, we just need to add the two columns together. The function used yields a nice sorted order.

​

However, DataFrames are a terrible lookup table, hence I have chosen to convert them into dictionaries

Although data cleaning is sparse in this project, it is compensated by the large amount of transformation of data in the EDA. This is caused by the data being scraped and concrete research into this niche area being rather lacking.

## EDA<a id='eda'/>

<div class="alert alert-block alert-info">

For each research questions shortlisted, outline your methodology in answering them. Discuss interesting observations or results discovered.

Please note to only show EDA that's relevant to answering the question at hand. If you have done any data modeling, include in this section.

</div>

### Q1. "Crosswordese"

Firstly, let us define "short answers" as anything with at most 7 letters, and "long answers" as anything with at least 8 letters

For each crossword, we will do the following:

1. Group all the clues from the specific day

2. Filter the DataFrame to only have short answers

3. For each puzzle, for each short answer that appears on the NGram table, check it against the corresponding year that it appeared, and add it to a total, call this number the "Score"

4. For each puzzle, for each short answer, compare it against the wordlist and give it the corresponding score, then add it all up for that outlet's daily crossword

We will obtain a DataFrame with the following: Day, Date, Outlet, Score, AScore, BScore

This is our base data for graphing.

This data is saved in "Q1 Data.csv"

Using the score, we can determine how much crosswordese is in it generally. The higher the score, the better the puzzle.

Then, we plot to observe any trends.

There is a limitation to this method. Some words are not found on the list and as such, I am unable to score them properly, hence I chose to give it a score of 0, as a baseline. This problem appears more when using the NGram dataset, as there are less entries and it is less extensive. However, it still provides a reasonably good image, as crosswords should be affected similarly by missing entries.

In the above graph, there does not seem to be much correlation between the day of week and how much crosswordese is present when we test it with wordlist A.

Somehow, when using the NGram dataset, which shows real life usage of these words, an interesting trend emerges. There seems to be 3 distinct sections of the stripplot. A more detailed discussion will be included in the results.

We can see that there is a slight trend upwards, showing improvement in the accessibility in terms of wordlist metrics. However, such a trend cannot be observed with NGram data.

The dip in 2000 can be explained by missing data.

# Q2 Freshness

<p> The proceedure to answer this question will be similar to Q1, where the wordlist will be used for comparison. For the wordlist score, I will just use it accordingly. However, for my own scoring of freshness of long answer, I will be scoring it on a harmonic scale, with the i th occurance of the answer having a score of 1/i. Then, to evaluate the score of the crossword, I will just sum it up.

Also, since long answers are rather rare in a crossword, counting them makes sense, so that will be taken into account too. <p>

<p> Afterwards, these points will just be plotted, to see if there are any trends to be spotted<p>

​

Through the week, we can see that the long score increases.

The max of the long score seems to be increasing.

In general, we can see that long score has been increasing, except in the case of Wall Street Journal, which has been decreasing.

There seems to be no correlation between month and the quality of crosswords. This is expected.

We can see that New York Times is the best for long answers, followed by LA Times and Wall Street Journal, then Universal and USA Today.

In a similar fashion, New York Times has the most long answers, followed by Wall Street Journal, then LA Times, then Universal, then USA Today.

No significant trend can be seen with this boxplot against time.

We can see that the long scoring between the two wordlists is similar and does not differ too much. It is thus reasonable to assume that changing the wordlist will not affect the results significantly. Hence our results are not that bad.

### Q3. Inclusive Clues

In this section, I will be analysing how inclusive clues are over the years.

I will first concatanate all the words in the crossword, then for each word, search for it in the namelist.

For each match, assign one point, then we can plot some trends.

The two namelists used will be from baby websites.

Of course, this method is limited by the namelist, however, with 1000 names for each gender, it should be fairly robust.

We can see that, in general, the number of names being used is decreasing. This may also be an effect of accessibility, trying to make it more about words than obscure celebrities. Wall Street Journal was removed from this comparison as it had values too high to scale the graph to be unreadable.